Skip to content

[Rust] Send TransformSession options as JSON - #8658

Open
jrozner wants to merge 2 commits into
Vector35:devfrom
jrozner:rust-transform-session-options
Open

jrozner wants to merge 2 commits into
Vector35:devfrom
jrozner:rust-transform-session-options

Conversation

@jrozner

@jrozner jrozner commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The five Rust TransformSession constructors (new, new_with_mode, from_view, from_view_with_mode, from_context) take a &Settings and send settings.serialize_schema() as the session options. That string is the settings schema (groups, titles, types, defaults), not the values set on the instance. The core expects a JSON object of setting keys and values, which is what Python (json.dumps(options)) and the C++ default ("{}") send. The core ignores the schema, so per-session overrides such as files.container.excludedTransforms have no effect from Rust.

On dev, with a ZIP of two files and an override instance that excludes the Zip transform:

global: transform="Zip" children=2
override: transform="Zip" children=2

The override should give transform="" children=0.

Change

new, new_with_mode, from_view, from_view_with_mode and from_context now take options: Option<O> with O: IntoJson, like load_with_options. None sends {}. They return Option<Ref<TransformSession>>: None if the options aren't a JSON object or the core returns no session (for example, before the core is initialized). The doc of new describes the options format and has an example.

This is a breaking change. Callers pass None::<&str> or Some(json) instead of &settings and unwrap the result. The only caller in this repository, test_context_transform, is updated. I chose it over keeping &Settings because it matches load_with_options and the Python and C++ APIs. Keeping &Settings would need a wrapper for BNSerializeSettings (SettingsAutoScope), and per-session options would then have to live in a named Settings instance that is shared process-wide and carries values between sessions.

Testing

New rust/tests/transform_session.rs builds a two-file ZIP at test time, so there is no binary fixture. test_session_options checks that with None, new_with_mode and from_view_with_mode extract the ZIP (root transform Zip, 2 children), and that with {"files.container.excludedTransforms": ["Zip"]} they don't (no transform, 0 children), and that options which aren't a JSON object ("", "not json", [], "Zip", null) give None. It passes on this branch.

The tests are in their own file because rust/tests/transform.rs registers MyDetectionTransform, which accepts any input of 4 bytes or more and would be applied to these sessions. On dev, running all of cargo test --test transform together already overflows the stack in test_context_transform for that reason; each test passes when run alone. This PR doesn't change that.

from_context sends the options too, but a session created from an existing context used that context's settings in my tests, so it isn't covered.

cargo fmt --check and cargo clippy -p binaryninja --all-features --lib --test transform --test transform_session are clean.

The `TransformSession` constructors took a `&Settings` and passed
`Settings::serialize_schema()` as the session options. That is the
settings schema (groups, titles, types, defaults), not the values set on
the instance, and the core expects a JSON object of setting keys and
values, which is what Python and C++ send. The core ignores the schema,
so per-session overrides such as `files.container.excludedTransforms`
had no effect from Rust.

`new`, `new_with_mode`, `from_view`, `from_view_with_mode` and
`from_context` now take `options: Option<O>` with `O: IntoJson`, the same
as `load_with_options`, and send `{}` for `None`. Like
`load_with_options`, they return `None` if the options can't be
converted to JSON.
@emesare
emesare self-requested a review October 15, 2026 20:28
@emesare emesare self-assigned this Oct 15, 2026
@emesare emesare added the Component: Rust API Issue needs changes to the Rust API label Oct 15, 2026
Comment thread rust/src/transform.rs Outdated
/// `options` are per-session setting overrides as a JSON object of setting keys and values, for
/// example `{"files.container.excludedTransforms": ["Zip"]}`. Pass `None` to use the settings as they are.
///
/// Returns `None` if `options` cannot be converted to JSON.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer this to be encoded in the type system, the "IntoJson" construct is kinda strange when it should be the callers responsibility to pass a valid json object (key value object). Not sure if this should be changed for this PR so its fine to leave like this, but something to keep in mind for future.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that a typed options parameter would be better. IntoJson is only implemented for strings and can't fail, so the "can't be converted to JSON" case I documented couldn't happen. 26a2c98 fixes that doc. I kept IntoJson here so these constructors match load_with_options. Switching both to a typed map or builder (serde_json is already a dependency) would make a good follow-up, so the two APIs change together.

Comment thread rust/src/transform.rs
options.as_ptr(),
)
};
assert!(

@emesare emesare Nov 21, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have removed the assertion that the session is always valid, AFAIK the only time that happens is if you call these functions outside a valid session (called function before the core has been initialized), which is totally possible, could be useful to document that, although a lot of functionality also fails when the core has not been initialized, and we do not document what those apis are.

We might want to have a list of rust apis that are "license checked" such that we can wrap a return type in an error state if the license is invalid, instead of just returning Option.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 26a2c98: new now says it returns None if the core can't create the session, for example because it hasn't been initialized. The other constructors point to new. I've left the license-checked Result idea for a separate, crate-wide change.

Comment thread rust/src/transform.rs Outdated
fn session_options<O: IntoJson>(options: Option<O>) -> Option<CString> {
match options {
Some(options) => Some(options.get_json_string().ok()?.to_cstr().to_owned()),
None => Some(c"{}".to_owned()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to validate this json object further?

JSON object of setting keys and values

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 26a2c98. session_options now parses the options as a JSON object (serde_json::Map<String, Value>), and the constructors return None for anything else (invalid JSON, arrays, strings, null). test_session_options covers those cases for new_with_mode and from_view_with_mode. It doesn't check the keys against registered settings; the core handles that.

Document when the constructors return `None`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Rust API Issue needs changes to the Rust API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants